You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR Commands — comment on this PR to trigger (requires collaborator approval):
Command
Description
/benchmark -f <framework>
Run every test the framework subscribes to
/benchmark -f <framework> -t <test>
Run one test only
/benchmark -f <framework> --save
Run and save results (updates the leaderboard on merge)
/benchmark -f <framework> -t <test> --save
Run one test and save results
/benchmark -f <framework> --compare <other>
Measure the deltas against another framework instead of this one
/benchmark-multiple -f <fw1>,<fw2>,...
Benchmark several frameworks in one run — takes -t and --save too; saved results land in a single commit
/benchmark-multiple --save
No -f needed: benchmark and save every framework the PR touches
/benchmark-test -t <test>
Benchmark all enabled frameworks subscribed to <test> and save the results
For /benchmark, always specify -f <framework>; the flags combine in any order. Results come back as a comment with a per-profile table of RPS, p99, CPU and memory — one table per framework on multi runs. A new benchmark comment while a run is in flight queues behind it (one deep) instead of cancelling it. For multi-framework PRs (dependency bumps, same-language refactors) prefer /benchmark-multiple, which runs everything in a single job and commits all saved results together, so no run overwrites another. --compare works on single-framework runs only.
What the deltas are measured against. By default, this framework's own results published on main - answering "did this change help?". When you are tuning a variant or a successor entry, --compare re-bases them on another entry instead:
The reply states which baseline it used, and profiles the other framework does not run show n/a rather than a delta.
Run benchmarks locally
You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.
40. Stopping all clients.Stopped all clients for thread #50
20. Stopping all clients.
finished in 5.06s, 741559.40 req/s, 2.47GB/s
requests: 3707797 total, 3740565 started, 3707797 done, 3707797 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 3708052 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 12.33GB (13240837391) total, 124.53MB (130580645) headers (space savings 76.48%), 12.15GB (13043457785) data
min max mean sd +/- sd
time for request: 56us 152.96ms 42.72ms 24.17ms 64.66%
time for connect: 18us 860us 355us 197us 63.09%
time to 1st byte: 1.00ms 107.50ms 18.90ms 19.12ms 86.43%
req/s : 633.95 877.45 724.08 39.44 68.65%
[info] CPU 4525.0% | Mem 528MiB
[run 3/3]
starting benchmark...
Application protocol: h2c
35.
42Main benchmark duration is started for thread #41.
.
39.
11.
57.
22.
34..
52Main benchmark duration is started for thread #34.
.
33.
.
.
.Main benchmark duration is started for thread #26.
35. Stopping all clients.
30Main benchmark duration is over for thread #29. Stopping all clients.
. Stopping all clients.
6. Stopping all clients.
30
10
. Stopping all clients.Stopped all clients for thread #46
finished in 5.06s, 734378.40 req/s, 2.44GB/s
requests: 3671892 total, 3704660 started, 3671892 done, 3671892 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 3672098 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 12.21GB (13112819026) total, 123.48MB (129483107) headers (space savings 76.45%), 12.03GB (12917183689) data
min max mean sd +/- sd
time for request: 54us 150.61ms 43.41ms 24.16ms 64.76%
time for connect: 18us 1.10ms 449us 239us 63.87%
time to 1st byte: 1.23ms 99.79ms 17.17ms 17.42ms 85.74%
req/s : 619.18 863.61 717.06 37.60 69.73%
[info] CPU 4557.8% | Mem 560MiB
=== Best: 732816 req/s (CPU: 4525.0%, Mem: 528MiB) ===
[info] saved results/json-h2c/1024/mq-bridge.json
httparena-bench-mq-bridge
httparena-bench-mq-bridge
==============================================
=== mq-bridge / json-h2c / 4096c (tool=h2load) ===
==============================================
[info] waiting for server...
[info] server ready
[run 1/3]
starting benchmark...
16.
.Warm-up phase is over for thread #17
.Warm-up phase is over for thread #.20
.Main benchmark duration is started for thread #Warm-up phase is over for thread #25.
.Main benchmark duration is started for thread #
48.
.
progress: Main benchmark duration is started for thread #20.84
% of clients started
2151.
.
13.
22.
56Warm-up phase is over for thread #Main benchmark duration is started for thread #10.27
28.
..
Application protocol: h2c
52. Stopping all clients.Stopped all clients for thread #
32
3. Stopping all clients.
3
. Stopping all clients.
28. Stopping all clients.
finished in 5.11s, 673277.60 req/s, 2.24GB/s
requests: 3366388 total, 3497460 started, 3366388 done, 3366388 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 3366734 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 11.18GB (12003855447) total, 113.19MB (118684191) headers (space savings 76.45%), 11.01GB (11824347815) data
min max mean sd +/- sd
time for request: 57us 823.71ms 185.97ms 136.78ms 56.52%
time for connect: 18us 6.23ms 2.39ms 1.69ms 56.81%
time to 1st byte: 5.85ms 573.20ms 67.98ms 69.59ms 89.43%
req/s : 98.95 257.27 164.30 19.81 69.73%
[info] CPU 4478.4% | Mem 1.1GiB
[run 2/3]
starting benchmark...
Application protocol: h2c
.Main benchmark duration is started for thread #56.
35.
29.
57.
0. Stopping all clients.
59Stopped all clients for thread #4
. Stopping all clients.Stopped all clients for thread #
15
33. Stopping all clients.
40
45
. Stopping all clients.
finished in 5.10s, 692741.40 req/s, 2.30GB/s
requests: 3463707 total, 3594779 started, 3463707 done, 3463707 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 3464011 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 11.50GB (12352643538) total, 116.82MB (122499671) headers (space savings 76.38%), 11.33GB (12167569125) data
min max mean sd +/- sd
time for request: 51us 779.30ms 178.71ms 140.15ms 57.74%
time for connect: 17us 7.92ms 3.04ms 2.33ms 54.05%
time to 1st byte: 3.42ms 571.94ms 37.58ms 42.57ms 91.41%
req/s : 92.77 262.68 169.05 21.80 68.58%
[info] CPU 4470.3% | Mem 1.5GiB
[run 3/3]
starting benchmark...
Application protocol: h2c
18.
18.
.Main benchmark duration is started for thread #57
.Main benchmark duration is started for thread #31.
30.Warm-up phase is over for thread #
59.
6.
15
. Stopping all clients.
36Main benchmark duration is over for thread #
46. Stopping all clients.
. Stopping all clients.
. Stopping all clients.
35. Stopping all clients.
finished in 5.09s, 672300.20 req/s, 2.23GB/s
requests: 3361501 total, 3492573 started, 3361501 done, 3361501 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 3361764 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 11.16GB (11987462475) total, 113.64MB (119162308) headers (space savings 76.32%), 11.00GB (11807565502) data
min max mean sd +/- sd
time for request: 60us 768.42ms 185.11ms 139.27ms 56.09%
time for connect: 17us 8.18ms 2.97ms 2.16ms 53.74%
time to 1st byte: 4.31ms 545.18ms 32.08ms 33.35ms 94.97%
req/s : 103.58 245.82 164.05 20.15 68.70%
[info] CPU 4475.0% | Mem 1.6GiB
=== Best: 679217 req/s (CPU: 4470.3%, Mem: 1.5GiB) ===
[info] saved results/json-h2c/4096/mq-bridge.json
httparena-bench-mq-bridge
httparena-bench-mq-bridge
[info] skip: mq-bridge does not subscribe to baseline-h3
[info] skip: mq-bridge does not subscribe to static-h3
[info] skip: mq-bridge does not subscribe to gateway-64
[info] skip: mq-bridge does not subscribe to gateway-h3
[info] skip: mq-bridge does not subscribe to production-stack
[info] skip: mq-bridge does not subscribe to unary-grpc
[info] skip: mq-bridge does not subscribe to unary-grpc-tls
[info] skip: mq-bridge does not subscribe to echo-ws
[info] skip: mq-bridge does not subscribe to echo-ws-pipeline
[info] skip: mq-bridge does not subscribe to echo-ws-limited
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/results/mq-bridge.json - 33 new, 39 total
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
httparena-postgres
httparena-redis
[info] restoring loopback MTU to 65536
.
45
53. Stopping all clients.
. Stopping all clients.Main benchmark duration is over for thread #
49Stopped all clients for thread #51
. Stopping all clients.
57
31. Stopping all clients.
39
18. Stopping all clients.
. Stopping all clients.
6
finished in 5.05s, 738456.80 req/s, 2.46GB/s
requests: 3692284 total, 3725052 started, 3692284 done, 3692284 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 3692486 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 12.28GB (13186805681) total, 125.33MB (131417704) headers (space savings 76.23%), 12.10GB (12988868727) data
min max mean sd +/- sd
time for request: 55us 137.09ms 43.26ms 24.19ms 64.80%
time for connect: 22us 1.07ms 395us 219us 61.13%
time to 1st byte: 1.09ms 106.07ms 15.70ms 14.30ms 82.71%
req/s : 636.32 862.35 721.05 37.69 66.70%
[info] CPU 4568.2% | Mem 511MiB
[run 3/3]
starting benchmark...
Application protocol: h2c
20.
.
33.
.
31.
.
8.
.
.
35.
57. Stopping all clients.
56. Stopping all clients.
finished in 5.06s, 711184.40 req/s, 2.37GB/s
requests: 3555922 total, 3588690 started, 3555922 done, 3555922 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 3556131 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 11.83GB (12699723476) total, 120.73MB (126597456) headers (space savings 76.22%), 11.65GB (12509061223) data
min max mean sd +/- sd
time for request: 53us 152.18ms 44.82ms 24.24ms 66.29%
time for connect: 17us 1.15ms 395us 223us 64.06%
time to 1st byte: 1.20ms 121.04ms 17.66ms 17.91ms 85.64%
req/s : 614.26 865.17 694.43 34.02 67.77%
[info] CPU 4531.0% | Mem 554MiB
=== Best: 731185 req/s (CPU: 4568.2%, Mem: 511MiB) ===
[info] saved results/json-h2c/1024/mq-bridge.json
httparena-bench-mq-bridge
httparena-bench-mq-bridge
==============================================
=== mq-bridge / json-h2c / 4096c (tool=h2load) ===
==============================================
[info] waiting for server...
[info] server ready
[run 1/3]
starting benchmark...
progress: Warm-up phase is over for thread #4633% of clients started
.
53.
.39Warm-up phase is over for thread #
1.
.
.2939Main benchmark duration is started for thread #Warm-up phase is over for thread #55.30
.Application protocol: Warm-up phase is over for thread #Warm-up phase is over for thread #h2c1442
...
.Main benchmark duration is started for thread #20.
.Main benchmark duration is started for thread #43.
.
.Main benchmark duration is started for thread #32
.Warm-up phase is over for thread #49.
11.
.
35.
62.
34Main benchmark duration is started for thread #54.
.
55. Stopping all clients.
43
48. Stopping all clients.
finished in 5.08s, 669848.60 req/s, 2.22GB/s
requests: 3349243 total, 3480315 started, 3349243 done, 3349243 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 3349475 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 11.12GB (11944636546) total, 113.73MB (119250076) headers (space savings 76.22%), 10.96GB (11764872701) data
min max mean sd +/- sd
time for request: 53us 860.10ms 186.09ms 137.35ms 56.40%
time for connect: 15us 7.85ms 2.62ms 2.12ms 61.01%
time to 1st byte: 3.67ms 518.83ms 63.23ms 65.33ms 90.28%
req/s : 92.15 264.69 163.46 20.07 69.31%
[info] CPU 4507.9% | Mem 1.1GiB
[run 2/3]
starting benchmark...
25.
.
progress: Main benchmark duration is started for thread #48.
93% of clients started
Application protocol: h2c
5.
2.
9.
.
.
1Stopped all clients for thread #43
13. Stopping all clients.
62. Stopping all clients.
20
finished in 5.11s, 670183.20 req/s, 2.23GB/s
requests: 3350916 total, 3481988 started, 3350916 done, 3350916 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 3351114 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 11.13GB (11950640437) total, 113.90MB (119431694) headers (space savings 76.19%), 10.96GB (11770665193) data
min max mean sd +/- sd
time for request: 56us 750.26ms 184.45ms 139.84ms 56.41%
time for connect: 16us 9.65ms 2.81ms 2.36ms 59.59%
time to 1st byte: 4.04ms 588.91ms 36.12ms 41.45ms 94.65%
req/s : 102.94 266.79 163.54 20.48 69.46%
[info] CPU 4482.9% | Mem 1.5GiB
[run 3/3]
starting benchmark...
.
46.
25Main benchmark duration is started for thread #46.
60Main benchmark duration is started for thread #18.
.Warm-up phase is over for thread #
.
.
44.Main benchmark duration is started for thread #60.
Application protocol: h2c
.
27.
61Stopped all clients for thread #62
. Stopping all clients.
21. Stopping all clients.
6. Stopping all clients.
finished in 5.11s, 682941.20 req/s, 2.27GB/s
requests: 3414706 total, 3545778 started, 3414706 done, 3414706 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 3414882 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 11.34GB (12179057533) total, 116.29MB (121940755) headers (space savings 76.15%), 11.17GB (11995425206) data
min max mean sd +/- sd
time for request: 52us 736.29ms 181.16ms 142.42ms 58.65%
time for connect: 13us 8.53ms 2.89ms 2.39ms 58.72%
time to 1st byte: 3.44ms 628.59ms 41.53ms 50.21ms 90.70%
req/s : 104.34 301.74 166.65 22.18 69.02%
[info] CPU 4496.0% | Mem 1.6GiB
=== Best: 668274 req/s (CPU: 4496.0%, Mem: 1.6GiB) ===
[info] saved results/json-h2c/4096/mq-bridge.json
httparena-bench-mq-bridge
httparena-bench-mq-bridge
[info] skip: mq-bridge does not subscribe to baseline-h3
[info] skip: mq-bridge does not subscribe to static-h3
[info] skip: mq-bridge does not subscribe to gateway-64
[info] skip: mq-bridge does not subscribe to gateway-h3
[info] skip: mq-bridge does not subscribe to production-stack
[info] skip: mq-bridge does not subscribe to unary-grpc
[info] skip: mq-bridge does not subscribe to unary-grpc-tls
[info] skip: mq-bridge does not subscribe to echo-ws
[info] skip: mq-bridge does not subscribe to echo-ws-pipeline
[info] skip: mq-bridge does not subscribe to echo-ws-limited
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/results/mq-bridge.json - 33 new, 39 total
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
httparena-postgres
httparena-redis
[info] restoring loopback MTU to 65536
Using mimalloc seems to had a significant improvement in json-comp for some reason.
Keeping it disabled - let me know if it is fine to enable it.
Edit: Checking again - mimalloc also significantly increased memory consumption. So - it was probably better for throughput, but not good for the final resulted score. Will evaluate - I think I will make a lib change to improve performance without mimalloc.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Updating mq-bridge to latest version.
PR Commands — comment on this PR to trigger (requires collaborator approval):
/benchmark -f <framework>/benchmark -f <framework> -t <test>/benchmark -f <framework> --save/benchmark -f <framework> -t <test> --save/benchmark -f <framework> --compare <other>/benchmark-multiple -f <fw1>,<fw2>,...-tand--savetoo; saved results land in a single commit/benchmark-multiple --save-fneeded: benchmark and save every framework the PR touches/benchmark-test -t <test><test>and save the resultsFor
/benchmark, always specify-f <framework>; the flags combine in any order. Results come back as a comment with a per-profile table of RPS, p99, CPU and memory — one table per framework on multi runs. A new benchmark comment while a run is in flight queues behind it (one deep) instead of cancelling it. For multi-framework PRs (dependency bumps, same-language refactors) prefer/benchmark-multiple, which runs everything in a single job and commits all saved results together, so no run overwrites another.--compareworks on single-framework runs only.What the deltas are measured against. By default, this framework's own results published on
main- answering "did this change help?". When you are tuning a variant or a successor entry,--comparere-bases them on another entry instead:The reply states which baseline it used, and profiles the other framework does not run show
n/arather than a delta.Run benchmarks locally
You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.
Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk) are built as self-contained Docker images on first run.